COMBINATORICS OF THE BASIC STRATUM 
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Abstract. We express the cohomology of the basic stratum of some unitary Shimura vari- 
eties associated to division algebras in terms of automorphic representations of the group in 
the Shimura datum. 



Introduction 

This article is a sequel to the article [T7] . We remove an hypothesis from the main Theorem 
of our previous article. In that article we proved a relation between the £-adic cohomology 
of the basic stratum of some simple Shimura varieties and the cohomology of the complex 
Shimura variety. These simple Shimura varieties are those of Kottwitz considered in his In- 
ventiones article [13J on the construction of Galois representation. The varieties are associated 
to certain division algebras over Q with involution of the second kind; we call such varieties 
Kottwitz varieties. 

When we proved the main theorem of our previous article we assumed (essentially) that the 
Newton polygon associated to the basic stratum had no integral point other than the begin 
point and the end point. In this article we solve the resulting combinatorial problems when 
one removes this simplifying condition from the theorem in case the prime p of reduction is 
split in the center of the division algebra defining the Kottwitz variety. 

A consequence of our final result is an explicit expression for the zeta function of the basic 
stratum of Kottwitz's varieties at split primes of good reduction. The expressions are in 
terms of: (1) Automorphic forms on the group G of the Shimura datum, (2) The determinant 
of the factor at p of their associated Galois representations, and (3) Polynomials in q a of 
combinatorial nature, associated to certain non-crossing lattice paths in the plane Q 2 . 

As an application we deduce a formula for the dimension of the basic stratum. 
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gave me many ideas and checked several times the arguments and proofs in this article. I also 
thank Laurent Fargues (my second advisor), and Olivier Schifmann, Peter Scholze, Alberto 
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Contents 



Introduction 
1. Notations 



2 



ARNO KRET 



2. Computation of some compact traces \a 

3. A dual formula 15 

4. Return to Shimura varieties 2C 

5. Examples 32 
References 35 



1. Notations 

Let p be a prime number and let F be a non- Archimedean local field with residue charac- 
teristic equal to p. Let wf G Of be a prime element, and define q := ^{Of/'cuf)- We write 
G n for the topological group GL n (F), and we write Ti(G) for the Hecke algebra of locally 
compact constantly supported functions on G. We often drop the index n from the notation 
if confusion is not possible. We call a parabolic subgroup P of G standard if it is upper 
triangular, and we write P = MN for its standard Levi decomposition. We write K for the 
hyperspecial group GL n (0p) and %q{G) for the Hecke algebra of G with respect to K. The 
group Pq C G is the standard Borel subgroup of G, T is the diagonal torus of G, and iVo is 
the group of upper triangular unipotent matrices in G. 

We write G, T, M, ... for the corresponding complex dual groups, G = GL n (C), T = (C x )™, 
and so on. If tt is an unramified representation of some Levi subgroup M of G then we write 
f m.tt £ M for the Hecke matrix of this representation. 

Let n be a positive integer. A partition of n is a finite, non-ordered list of non-negative 
numbers whose sum is equal to n. A composition of n is a finite, ordered list of positive 
numbers whose sum is equal to n. Recall that the compositions of n correspond to the 
standard parabolic subgroups of G. 

We write A for the ring C[Xf l ,Xf l , X^ 1 ] 6 ™. The Satake transform S provid es an 
isomorphism from ~Hq(G) onto the ring A. 

Let n and a be positive integers, and let s be a non-negative integer with s < n. We call 
the number s the signature, and we call the number a the degree. The function f nas G 7io(G) 
is the spherical function whose Satake transform is 

(1.1) q as{n-s)/2 ^ = q as(n-s)/2 ^ Xf G A. 

v£& n -Hs IC{l,...,n},#J=a iel 

We put f nas = when n,a,s G Z>o are such that n < s. We will call / nQS a simple Kottwitz 
function. The composite Kottwitz functions f nacj are obtained from partitions cr of s as follows. 
Let a = (o"i, o"2, . . . , o~ r ) be a partition of s. Then we write f naa G %q(G) for the convolution 
product f naai * f naa2 * • • • * f naUr G Uq{G). 

We write xj? f° r the characteristic function on G of the subset of compact elements. Let 7r 
be a smooth G-representation of finite length and / a locally constant, compactly supported 
function on G. Then we write Tr(x^/, 7r) for the compact trace [7] of / against 7r. 
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Let m,m' £ Z>i. If tt (resp. tt') is a smooth admissible representation of G m (resp. 
G m i), then we write tt x tt' for the G m+m i -representation parabolically induced (unitary 
induction) from the representation tt ® tt' of the standard Levi subgroup consisting of two 
blocks, one of size m, and the other one of size m'. The tensor product tt ® tt' in the above 
formula is taken along the blocks of this Levi subgroup. We write 1Z for the direct sum 
©nez> Groth(G n ) with the convention that Go is the trivial group. The group Go has 
one unique irreducible representation uq (the space C, with trivial action). The operation 
"direct sum of representations" together with the product "x" turns the vector space 1Z into 
a commutative C-algebra with ctq as unit element. We call it the ring of Zelevinsky. 

The ring of Zelevinsky has an involution t, called the Zelevinsky involution. Aubert [1J gave 
a refined definition of this involution, making sense for all reductive groups. The involution 
is defined by X 1 := ^2p = mn £ P^- n( ^p(^N(Sp 1 ^ 2 )) for all X E 7Z. With 'involution' we mean 
that l is an automorphism of the complex algebra 1Z and it is of order two: l 2 = Idn. 

We write v for the absolute value morphism from GLi(i ? ) = F x to C x . By a segment 
S = {x, y) we mean a set of numbers {x, x + 1, . . . , y} where x, y € Q and where we need to 
explain the conventions in case y < x. In case y is strictly smaller than x — 1, then {x, y) = 0; 
in case x is equal to y, then the segment (x, y) = {x} has one element. We have one unusual 
convention: For y = x — 1 we define the segment {x, y) to be the set {*} of one element 
containing a distinguishing symbol The length £(S) of a segment S = (x, y) is defined to 
be y — x + 1. Thus the segment {*} has length 0, the segment {x} has length 1, the segment 
{x, x + 1} has length 2, etc. We put £(x, y) = —1 in case y < x — 1. 

For any segment {x,y) with y > x we write A{x,y) for the unique irreducible quotient of 
the induced representation v x x v x+1 x • • • x u y . We define A{*} to be a"o (the one-dimensional 
representation of the trivial group GLq(F)), and we define A(x,y) to be in case y < x — 1. 
For any segment S of non-negative length the object AS is a representation of the group 
GL n (F), where n is the length of S. 

For the standard properties of segments we refer to Zelevinsky 's work [30] (cf. |26|). but 
note that our conventions are slightly different, because we allow rational numbers in the 
segments and we have the segment {*}. We mention that this difference is there only for 
notational purposes, and that it does not change the mathematics. 

For any finite ordered list of segments S\, S2, ■ ■ ■ , St we have the product representation 
tt := (ASi) x (AS2) x • • • x (ASt). Observe that, due to our conventions, in case S a = {*} 
for some a, then AS a is the unit in 1Z, and 

(1.2) tt = (ASi) x (AS 2 ) x • • • x (ASa) x • • • x (AS t ) & K, 

where the hat means that we leave the corresponding factor out of the product. In case Sb = 
for some index 6, then we have tt = in 1Z. 

In the combinatorial part of this article the representations of interest are the Speh 
representations. We recall their definition here. Let t, h be positive integers such that 
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n = th. We define Speh(/i, t) to be the (unique) irreducible quotient of the representation 
t-i l-t 

Stc h v 2 x • • • x Sta h v 2 • This representation has t segments, S a = (x a ,y a ), a = L • • • ,t, 
where 

t — h, , t + h 
x a = — (a - 1) and y a = — a. 

Observe that, for each index a, we have £S a = h. Furthermore, for each index a < t, we have 
x a +i =x a -l and y a +i =y a -l. 

If P = MN C G is a standard parabolic subgroup of G, then we have the spherical functions 
XNiXN hi T~(-q{M) associated to the acute and obtuse Weyl chambers. We refer to [T7, Eq. 
(1.1), (1.2)] for the precise definition and explicit description of these functions. 

2. Computation of some compact traces 

In this section we compute the compact traces Tr(x^ fnas, ^) of the simple Kottwitz func- 
tions / on a certain class of representations ir. This class will be sufficiently large to contain 
all smooth representations that occur in the cohomology of (basic) strata of unitary Shimura 
varieties at primes of good reduction. 

We will follow the following strategy to compute Tr(xf fnas, tt)- A semistable representation 
7r of G is called standard if it is isomorphic to a product of essentially square integrable 
representations. The computation of the compact trace Tr(x^ fnas, tt) on a square-integrable 
representation is easy, and using van Dijk's formula adapted for compact traces [17, Prop. 1.5], 
we easily deduce formulas for compact traces on the standard representations. Any semistable 
irreducible representation it may b ^ (uniquely) written as a sum 7r = J2i ■ I & TZ where 
/ ranges over the standard representations, and the coefficients c/ € C are for nearly all /. 
We have 

Tr( X ?/,7r) = ^ C/ Tr(x?/,7). 

Thus, there are two steps to compute Tr(x^/, it): (Probl) Know the coefficients c/ and 
(Prob2) Make the sum Y2i c i Tr(Xc /> -0- The first problem (Probl) is related to the Kazhdan- 
Lustzig conjectur^. The "Kazhdan-Lustzig Theorem" of Beilinson-Bernstein [3] (and |llj ) 
interprets the multiplicity of any given irreducible representation it in the representation 
/. The Kazhdan-Lustzig Theorem interprets this multiplicity as the dimension of certain 
intersection cohomology spaces, and also as the value at q = 1 of certain Kazhdan-Lustzig 
polynomials. 

For the irreducible representations ir contributing to the cohomology of Newton strata of 
unitary Shimura varieties we will not have to deal with problem (Probl). The Theorem of 
Moeglin-Waldspurger |23| (cf. [171 Thm. 2.1]) for the discrete spectrum of the general linear 
group implies that these representations must be of a very particular kind (rigid representa- 
tions, cf. [loc. cit, p. 14]). Any rigid representation is a product of unramified twists of Speh 

^Zelevinsky proved in [30] that the standard representations form a basis of 1Z as complex vector space. 
2 This conjecture is a Theorem, see [j3 Thm. 8.6.23] 



COMBINATORICS OF THE BASIC STRATUM 



5 



representations in 1Z, and therefore we restrict our attention to these Speh representations 
only. Tadic has solved the first problem (Probl) for the Speh representations. The coeffi- 
cients c/ turn out to be —1, or 1 for these representations (precise statement in Theorem l2.ip . 
Therefore, we are mostly concerned with the second problem (Prob2). 

2.1. Tadic's determinantal formula. We recall an important character formula of Tadic 
for the Speh representations. This formula is a crucial ingredient for our computations. 

Let Si = (xi,yi),S2 = (£2,2/2), • • • , St = (xt,yt) be an ordered list of segments defining a 
representation of the group G = GL n {F). Let &t be the symmetric group on {1,2,... , t}. 
For any w E &t we define the number n™ to be y a — x w r a ) + 1. We have 

k / k \ / k \ / k \ / k \ k 

(2.1) Y<<= - \ J2 x Ma) )+ k = E^ - E 2 ^ + fc = J>« = n. 

a=l \a=l / \a=l / \a=l / \a=l / a=l 

The numbers n™ need not be positive. We define & t C &t to be subset consisting of those 
permutations w £ &t such that the numbers are positive or 0. If the permutation w lies 
in the subset & t C &t, then (n„) is a composition of n. Assuming that w € 6^ we will write 
P w = M W N W for the parabolic subgroup of G corresponding to the composition (n^). 

Let w € & t . We define the segments Sf := (x w ^,y\), Sf := (x w r 2 \, 1/2), • • • , Sf := 
{ x w(t)-,Ut)- We have 1{S™) = n„. We let A w be the representation of M ffi defined by 
(AS 1 ™) ® • • • <8> (AS 1 ^ ), where the tensor product is taken along the blocks of M w . The 
representation /„, is defined to be the product ASf x AS 2 x • • • AS™, i.e. it is the (unitary) 
parabolic induction Indp^A,,, of A w to G. In case w S &t\& t we define both A w and I w to 
be 0. 

Remark. It is possible that S™ = {★} for some permutation In that case the representation 
AS™ is the unit element gq of 1Z, and thus can be left out of the product that defined /,„ (cf. 
Equation (jl~2"]) ). 

In these notations we have the following theorem: 

Theorem 2.1 (Tadic). Let ir be a Speh representation of G and let S\ = {x\,yi), S2 = 
(^2,2/2), • • • St = (xt, yt) be its segments. The representation it satisfies Tadic's determinantal 
formula 

7T = sign(w)/ w . 

U)G6t 

Proof. This Theorem was frst proved by Tadic in |29] for Speh representations with a difficult 
argument. Chenevier and Renard simplified the proof and observed that the above expression 
is a determinant of a matrix with coefficients in Zelevinsky's ring 1Z. Also Badulescu gave a 
simpler proof of Theorem 12.11 in the note [2] using the Moeglin-Waldspurger algorithm [22] . 
Recently Lapid and Minguez [201 Thm. 1] extended the formula to the larger class of ladder 
representations. □ 
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Remark. Our formulation of Theorem 12.11 is weaker than the theorem proved by the above 
authors, because we consider only semistable Speh representations. (They have a similar 
statement also for the non semistable Speh/ladder representations.) 

By the definition of the subset & t C &t we have for all w G ©t that I w 7^ if and only if 
w € 6j, and thus we may as well index over the elements w E & t in the sum in the above 
Theorem. In the cases where the inclusion & t C &t is strict, the subset & t is practically 
never a subgroup of &t, it will neither be closed under composition nor contain inverses of 
elements. 

2.2. Lattice paths and the Steinberg representation. In this section we will express 
the compact trace of the functions f nas on the Steinberg representation in terms of certain 
lattice paths in Q 2 . 

We fix throughout this section a positive integer a, called the degree. This integer will 
play only a minor role in the computations of this section as it affects only the weights of the 
paths. The degree will become more important later. 

Let A + be the polynomial ring C[g a |a G Q] of rational, formal powers of the variable q. 
Equivalently, A + is the complex group ring C[Q + ] of the additive group Q + underlying Q. 
A path L in Q 2 is a sequence of points vo,v\,V2, ■ ■ ■ ,v r such that Vi + \ — V{ = (1,0) (east), 
or Vi + \ — Vi = (1,1) (north-east). The starting point of L is vo and the end point is v r ; 
the number r is the length. An eastward step (1,0) has weight 1 and a north-eastward step 
(a, b) — > (a + 1, b + 1) has weight q~ a a € A + . The weight of the path L is defined to be the 
product in A + of the weights of its steps. 

Remark. We allow paths of length zero; such a path consists of one point vq and no steps. 
The weight of a path of length is equal to 1. The paths of length correspond to compact 
traces on the special segments {*} introduced earlier. 

Let L be a path in Q 2 . Connect the starting point vq of L with its end point v r via a 
straight line i. Then L is called a Dyck path if all of its points v a lie on or below the line £ 
in the plane Q 2 . The Dyck path is called strict if none of its points v a other than the initial 
and end point, lies on the line i. 

Let x, y be two points in Q 2 . Then we write Dyck s (x, y) S A + for the sum of the weights 
of all the strict Dyck paths that go from the point x to the point y. We call the polynomial 
Dyck s (a;, y) the strict Dyck polynomial. There are also non-strict Dyck polynomials Dyck(x, y) 
but we are not concerned with those in this subsection; they are important for the computation 
of compact traces on the trivial representation. 

Let / 6 T-Lq(G) be a function. We abuse notation and write XN<S(f) for the T-Satake trans- 
form of the function XnJ^ ^- This truncation xn! of an element / € A is best understood 
graphically. 

We first extend the notion of a path slightly to the concept of a graph. A graph in Q 2 
is a sequence of points Vq, Hi, . . . , v r with v i+ i — = (1, e), where e is an integer. Thus the 
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paths are those graphs with e € {0, 1} for each of its steps. We define the weight of a step 
(a, b) — > (a + 1,6 + e) to be q~ a e a g A + , and the weight of a graph is the product of the 
weights of its steps. 

To a monomial X = X{ X X% ■ • • X%> € C[Xf l ,X± l , X± L ], with e,eZ and £" =1 e { = s 
we associate the graph Qx with points 

(2.2) t7 := ^(^), # := v + (i, e n + e n _i + . . . + e n+ i_i) G Q 2 , 

for i = 1, . . . ,n. If x € Q, then we write £{x) for the point (x, ^x) on the line Because the 
sum Y^i=i e i ls e Q ua l t° s ) the en d point of the graph is 

^) + (n, S ) = ^+l)eQ 2 . 

We have xn X = X if and only i^ 

(2.3) e a + e 2 + . . . + <h > f i, 

for all indices % < n (and if xn X ^ X, then xn M = 0). The condition in Equation ()2.3[) 
is true if and only if the graph defined in Equation (|2.2|) lies strictly below the straight line 
£ C Q 2 of slope ^ going through the origin. Furthermore, the evaluation of X at the point 

( 1— n 3— n n— 1 \ ^ 

9 2 ,9 2 2 J ET, 

equals the weight of the graph Qx- 

Remark. The reader might find it strange that in Equation (|2.2p we let the graphs go in the 
inverse direction. Why did we make this convention? We made this convention because we 
want graphs that stay below the line t. If we draw the graphs using the 'natural' formula, 
then we get a graph above the line £ going from right to left. So why not consider only graphs 
that stay above 11 Of course this is equivalent, but later, when we compute the graph for the 
trivial representation, we get get graphs whose 'natural' formula stays below £ and goes from 
left to right. Thus, either way, we have to invert directions. 

1/2 

Lemma 2.2. Consider the representation ir = ~\-t(&p ) of the group T . Let f be a function 
in the spherical Hecke algebra of T. Then the trace of f against tt is equal to the evaluation 
of S{f) E A at the point 

( l—n 3— n n—1 \ ^ 

q 2 ,q 2 2 ) E T, 

Proof. The character 8 l p^ on T is equal to 



n— 1 n— 3 n—1 

T3 (h,t 2 ,...,t n ) .— ► |ti| 2 \t 2 \ 2 ...\t n \ 2 G 



•'This is true because the fundamental weights of the general linear group are of the form Hi + ■ ■ ■ + 
Hi — ^7 (-Hi + Hi + . . . + H„) on ao. The statement follows also directly from the conclusion made at Equation 
(1.10) in the proof of Proposition 1.11 in [IT] . 
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To any (rational) cocharacter v G X*(T) we may associate the composition (5p o o u) : F x — > 
T — > C x . We evaluate this composition at the prime element wp G F x . Thus we have an 
element of the set 

(2.5) Hom(X,(T),C x ) = Hom(X*(f ), C x ) = X*(f)® z C x = f (C), 

where the last isomorphism is given by 

X*(T) ® Z C X 9^^ u(z) G f (C). 

We have T = (F x ) n and thus we have the standard basis ej on X*(T). This corresponds to 
the standard basis e% on X*(T) via the first two equalities in Equation (I2.5p . If we take v = a 

1/2 

in (<5 p ^ o z/)(otf) then we get 

1 lo n-1 . . 1— n . , 

(4f o e ,,)( roF ) = — -+ 1 = 

This completes the verification. □ 

Let I C Q 2 be the line of slope f- through G Q 2 that we introduced earlier. We write £(x) 
for the point (x, ^x) on £ if x G Q. 

Lemma 2.3. The compact trace Tr(Xc fnas, Stc) on t/ie Steinberg representation is equal to 
the polynomial (_l)»-ig«("-»)/2 . Dyck s (^(±^), + 1)) G A+. 

Proof. The proof is a translation of a result that we obtained in our previous article: 

(2-6) TKxf/,St G ) = (-l^Tr fev /n QS , lr(^ /2 ' ' 



1/2 1/2 

(see [T71 Prop. 1.13]). NB: We wrote 5p Q and not 5 Pq ; the additional sign is there because 
the Jacquet module at Po of the Steinberg representation Sto is lr(<5p ). 

In case / = f nas then every monomial X occurring in <S(/) is multiplicity fre^, and 
therefore the graph Qx is in fact a path. The above construction X \- > Qx provides a bijection 
between the monomials that occur in S(f) and the possible paths that go from the point vq 
to the point v r . Finally Xn X ^ if and only if the corresponding path is a Dyck path (see 
Equation (|2.3p ). This completes the proof. □ 

Compact traces are compatible with twists: 

Lemma 2.4. Let x be an unramified character of F x , it a smooth irreducible G representa- 
tion, and f naa G %q{G) a function of Kottwitz. Then Tr(x^ f naa , tt®x) = xfajfl'^iXc f^)- 

Proof. pH Lem. 1.8]. □ 

Lemma 2.5. Assume that tt is an essentially square integrable representation of the form 
AS, where S = (x,y) is a segment of length n. Then 

s(n—s) 

Tr( Xc G /, A(x,y)) = (-l)"" 1 • q^~ ■ Dyck s (^),% + 1)). 



^Multiplicity free in the sense that no variable Xi occurs with exponent ej > 1 in X. 
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Proof. The representation (AS) ®v 2 is the Steinberg representation, and so Lemma 
applies to it. The result then follows from Lemma 12.41 □ 

2.3. Lattice i-paths and standard representations. We describe the compact traces on 
the standard representations of G using "t-paths". 

Let t be a positive integer. Let x = (x a ) and y = (y a ) be two ordered lists of points 
in Q 2 , both of length t. A t-path from x to y is the datum consisting of, for each index 
a € {1, 2, . . . , t}, a path L a from the point x a to the point y a . A t-path (L a ) is called a Dyck 
t-path if all the paths L a are Dyck paths. The Dyck path (L a ) is called strict if, for each 
index a, no point V{ of L a other than vq and v r lies on the line t. The weight weight(L a ) 
of a t-path (L a ) is the product of the weights of the paths L a , where a ranges over the set 
{1,2, . .. ,t}. We extend the definition of the strict Dyck polynomial Dyck s (x, y) £ A + also 
to t-paths: The polynomial Dyck s (x, y) £ A + is by definition the sum of the weights of the 
strict Dyck t-paths from the points (x a ) to the points (y a )- We have 



(2.7) Dyck s (f , y) = \\ Dyck s (x a , y a ) € 



a=l 

Lemma 2.6. Let Si = (x%, y\), S2 = (X2, y2)i ■ ■ ■ j St = (xt,yt) be a list of segments and let 
I be the representation (AS\) x (AS2) X ••• X (ASt). Then the compact trace Tr(xf fnas, I) 
is equal to (— l) n ~*Dyck s (x, y), where for the indices a = l,...,t we have x a := £(x a ) and 
y a :=£(y a + l)- 

Remark. The sign (— l) n ~* is equal to £MnP 0; where M is the standard Levi subgroup of G 
corresponding to the composition $^0=1 £(n a ) of n. 

Proof. Let P be the parabolic subgroup of G corresponding to the composition n = 
EU^a) of n. Let 

Xm De * ne characteristic function on M of the subset of elements 
m G M such that (zu^ , il^(m)} = for all a G Ap. By the integration formula of van Dijk 
for compact traces |171 Prop. 1.5] we have 

Tr( X °f, I) = Tr (x^ fZl (A5i) x (AS 2 ) x • • • x (AS t )) 
(2-8) = Tr ( X f X %f£l (ASi) x (AS 2 ) x • • • x (AS t 

In our previous article we proved (T7J Prop. 1.10] that the function xf fnol is equal to 

(2-9) q ^ °" a fnasj ® fnctS2 ® ' ' ' ® fnasti 

where s a := ^f-s, and 

s(n - s) ^ S a( n a ~ S a ) 
a=l 



C(n a ,s a ) := 2^ 
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The constant term in Equation (|2.9[) vanishes in case one of the numbers s a is non-integral. We 
have (x^/na2)*- P ° nA/ ^ = X%fnas ■ Consequently, one may rewrite the trace in Equation (|2.8p 
to the product 

t 

q aC ^ s) l[Tr( X ^f naaSa ,AS a ), 
a=l 

By Lemma 12.51 we obtain 

g aC(n a ,s) JJ ( _ 1) n a -l g naK 2 - Sa) -Dyck s (^(x a )^(y a + 1)). 
a=l 

Note that the condition that s a is integral precisely corresponds to the condition that the 
vertical distance between the point y a and the point x a has to be integral before paths can 
exist. Therefore the expression in this last Equation simplifies to the one stated in the Lemma 
and the proof is complete. □ 

2.4. Non-crossing paths. We express the compact traces on Speh representations in terms 
of non-crossing lattice paths. 

We call a t-path (L a ) crossing if there exists a couple of indices a, b with a ^ b such that the 
path L a has a point v € Q 2 in common with the path Lb- There is an important condition: 
• The point v of crossing must appear in the list of points v a i that define L a and it 
must also occur in the list of points Vbi that define L&. 

(Because we work with rational coordinates, the point of intersection could be a point lying 
halfway a step of a path (for example). We are ruling out such possibilities.) 




Figure 1. An example of a 3-path corresponding to the representation ir of 
GL 54 (F) defined by the segments (3,20), (2,19) and (1,18). We take s = 27 
and we take the permutation w = (13) € 63. The 3 dots on the lower left 
hand corner are the points xi,x 2 and X3 in Q 2 respectively; the points yi,y 2 
and 2/3 are in the upper right corner. Observe that this 3-path is non-strict. 

We write Dyck+(x, y) for the sum of the weights of the non- crossing strict Dyck t- 
paths. Let tt be the Speh representation of G associated to the Zelevinsky segments 
(x 1 ,yi),(x 2 ,y 2 ),...,(x t ,y t ) with x\ > x 2 > ■ ■ ■ > x t and y x > y 2 > . . . > yt- We define 
the points x a := £(x a ) G Q 2 and y a := £(y a + 1) G Q 2 , for a = 1, 2, . . . , t. The group &t acts 
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on the free Q 2 -module Q 2t = (Q 2 )* by sending the a-th standard basis vector e a G (Q 2 )' to 
the basis vector e w i a \ G (Q 2 )*. Thus if we have the vector x G Q 2 *, then we get the new vector 
x 1 " whose a-th coordinate G Q 2 * is equal to w(a)-th coordinate of the vector x. 

Remark. The difference ^ • (y a + 1) — f- ■ x a need not be integral. In that case there do not 
exist paths from the point x% G Q 2 to y a G Q 2 . 

Let 7T be a Speh representation of type (h,t). The points x a G Q 2 and y a G Q 2 lie on 
the line i C Q 2 , and the point x a lies on the left of the point y a with horizontal distance 
y a + 1 — x a = l(S a ) = h. The two lists of points may overlap: There could exist couples 
of indices (a, 6) such that x a = j/j,. All points x a and yj, are distinct if we have h > t (cf. 
Figure 1). 

Assume h > t. Then, because all the points x a , y b are distinct, there is no permutation 
w G &t such that one of the segments = (x w ^,y a ) is empty or equal to {*} for some 
index a. In particular we have & t = 6f 

Definition 2.7. To any point v G Q 2 we associate the invariant p{v) := P2(v) G Q/Z where 
P2 '■ Q 2 — > Q is projection on the second coordinate. 

Remark. The horizontal distance between the point Xb and the point y a is integral for all 
indices. Therefore the invariant of the first coordinate is not of interest. However, the vertical 
distance is the number = ^n" G Q, which certainly need not be integral. 

Using this invariant we define a particular permutation wq (z &t'. 

Definition 2.8. Assume h > t and assume that for each invariant p G Q/Z the number of 
indices a such that the point x a has invariant p is equal to the number of indices a such that 
the point y a has invariant p. The element wq G &t is the unique permutation such that for 
all indices a, b we have 
(2.10) 

< b and p(x a ) = p(x b )^j =^ fu^ta) > and P(Va) = P(Vb) = p(x a )^j ■ 

Remark. Observe that the permutation wq depends on the integer s because the heights of 
the points x a , y a , and therefore also their invariants depend on s. 

Remark. If our assumption on the invariants p(x a ) and p(y a ) in Definition 12.81 is not satisfied, 
then the permutation wq cannot exist because it has to induce bijections between sets of 
different cardinality. 

One could also define the permutation wq G &t inductively: First the index w^ 1 ^) G 
{1, 2,3, ... ,t} is the minimal index b such that the points xt and y% have the same invariant. 
Next, the index Wq 1 ^ — 1) G {1, 2, 3, ... ,t} is the minimal index b, different from Wq 1 ^), such 
that x a and y^ have the same invariant. And so on: Wq 1 (t — {) G {1, 2, 3, . . . , t} is the minimal 
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index b different from the previously chosen indices w (t), w (t — 1), ... , w (t — i + 1), 
such that the points and xt-i have the same invariant. 

Lemma 2.9. Let ir be a Speh representation with parameters h,t with h > t. Let d be 
the greatest common divisor of n and s and write m for the quotient t. Define the points 
x a := £(x a ) and y a := £(y a + 1). Let d be the greatest common divisor of n and s, and write 
m for the number 5 € Z. The following two statements are equivalent: 

(i) for each invariant p £ Q/Z the number of indices a such that the point x a has invariant 
p is equal to the number of indices a such that the point y a has invariant p; 

(ii) m divides t or m divides h. 

Remark. The number m is the order of the element - in the torsion group Q/Z. 
Proof. We first claim that u m\t (i) n . We have 

(2.H) P {Xa+l) = P{x a ) ~ f G Q/Z 

and the same relation for the points y a . Therefore, if m divides t, then the possible classes 
of the points x a are equally distributed over the subset ^Z/Z C Q/Z, and every invariant 
occurs precisely ^- times. The same statement also holds for the points y a , and in particular 
(i) is true. This proves the claim. 

We now claim that u m\h => (i)". Assume m\h. Then the invariants of x a and y a are the 
same for all indices a. Thus (i) is true. 

We prove that "(m/i and mj(h) =>■ ((i) is false)". Assume mj(t and m/h. We first reduce 
to the case where t < m. Assume t > m. Consider the elements 

(2.12) p(xi),p(x 2 ),...,p(x m ), and p(y x ), p(y 2 ), . . . , p(y m ) G Q/Z. 

By Equation (12.1ip every possible class in ^Z/Z occurs precisely once in both lists. Thus, the 
truth value of (£) is not affected if wg r6movG th.6 GlGincnts Xi, X2 ; ■ ■ ■ 5 %m 

and 2/1,2/2, • • • ,y m 

from the respective lists. Renumber the indices and repeat this argument until t < m. Because 
we assumed that t did not divide m there remains a positive number of elements in the list 
x a and 27a- We renumber so that the indices range from 1 to t. Then we have reduced to the 
case where 1 < t < m. 

Now look at the two lists p(x*i), p{x2), ■ ■ ■ , p(x*t) and p(yi), p(jj2), ■ ■ ■ > p(yt)- I n both lists 
every class in Q/Z occurs at most once. We assumed that m does not divide h, and therefore 
p(x*x) ^ p(yi). If there does not exist an index b such that p{x\) = p{yb), then (i) is false and 
we are done. Thus assume p{x\) = p(yb) for some 1 < b < t. By Equation (12. lip we then 
have p(yb~i) = piyb) + f = p{%i) + f • The invariant £ := p{x\) + ^ G Q/Z does not occur in 
the list p(xi), p(x2), ■ ■ ■ ,p(xt). Thus, we have found an invariant, namely £, occurring once 
in the list of invariants of the elements y a and does not occur in the list of invariants of the 
elements x a . This contradicts (i) and completes the proof. □ 
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Theorem 2.10. Let ir be a Speh representation with parameters h,t with h > t. Let d be 
the greatest common divisor of n and s and write m for the quotient 5- Define the points 
x a := £(x a ) and y a := £(y a + 1). The compact trace Tr(xf fnas, tt) on tt is non-zero if and 
only if m divides t or m divides h, and if the compact trace is non-zero, then it is equal to 

s(n—s) 

(—l) n ~ t sign(wo)q 2 "Dyckt" (af^° , y ) , where the permutation wq G &t depends on s and is 
defined in Definition \2.8[ 

Remark. Perhaps one could extend this Theorem to obtain formulas for compact traces on 
Ladder representations as considered by Minguez and Lapid in [20] . 

Proof of Theorem \2.1(K For a technical reason we assume that < s < n. In case s = we 
have f nas = lcL n (o F )- All the elements in GL n (0p) are compact and therefore x^fnao = 
fnao- The compact trace becomes the usual trace and the theorem is easy. A similar argument 
applies in case s = n. Thus we may indeed assume < s < n. 

By Theorem 12.11 the compact trace Tr n) is equal to the combinatorial sum 
SweSi s i§ n ( u; ) Tr(x^/) Iw) for any Hecke operator / € %{G). We apply it to the Kottwitz 
functions / = f nas . We have & t = &t because h > t. Let w € &f Recall that van Dijk's 
formula is also true for truncated traces |17} Prop. 1.5], and thus for any w 6 &' t the trace 
Tr(xff,I w ) equals Tr(x^ f^ Pw \ A w ). Thus we have the formula 

(2.13) TV( Xc G /,vr) = ]T signH • Ti( X ^f^\ A w ). 
By Lemma [2761 we get for / = f nas , 

(2.14) Tr(xc /, t) = Q Sj ^ a E ' £p °™ w • Dyck s (^, y). 

We apply a standard combinatorial argument Put the lexicographical order < on Q 2 : 

Vu, v € Q 2 : (u < v) (ui < «i or (ui = i?i and U2 < ^2)). 

Let (L a ) be a strict Dyck £-path from the points f" to the points y, and assume that (L a ) 
has at least one point of crossing. Let v G Q 2 be the point chosen among the points of 
crossing which is minimal for the lexicographical order on Q 2 . Let (a, b) a couple of different 
indices, minimal for the lexicographical order on the set of all such couples, such that v lies 
on the path L a and also on the path Lf,. We define a new path L' a , defined by following 
the steps of Lf, until the point v and then following the steps of the path L a . We define Li 
by following L a until the point v and then continuing the path Lb. For the indices c with 
c / a, b we define L' c := L c . Observe that (L' a ) is a t-path from the points x^ ab ' w to the 

^The Lindstrdm-Gessel-Viennot Lemma. The argument appears in many (almost) equivalent forms in the 
literature. We learned and essentially copied it from Stanley's book 28, Thm 2.7.1]. Note however that, strictly 
speaking, the Theorem 2.7.1 there does not apply as stated at this point in our argument. In the paragraph 
that follows we show that Stanley's argument may be adapted so that it does apply to our situation. 
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points y. Furthermore, it is a Dyck path (with respect to this new configuration of points), 
and we have weight(L a ) = weight(L' a ) because the weight is the product of the weights of 
the steps, and only the order of the steps has changed in the construction (L a ) i-> (L' a ). The 
construction is self-inverse: If we apply the construction to the path (L' a ) then we re-obtain 
(L a ). Both paths (L a ) and (L' a ) occur in the sum of Equation (I2.14p , The sign £p nAf„ is 
equal to (-l)«- 1 (-l)*- 1 (-l)#{ ce { 1 . 2 '-.*}|^(c)=jr }_ By the assump tion that h>t, the points 

in the list x are all different to the points in the list y, and therefore the sign £p nM„ equals 
(— l) n_ * (and does not depend on the permutation w). The sign of the permutation w is 
opposite to the sign of (ab)w. Consequently, the contributions of the paths (L a ) and (L' a ) to 
Equation (|2.14j) cancel, and only the non-crossing paths remain in the sum. We find 

s(n— s) 

(2.15) Tr( Xc G /,vr) = (-lf-'g 2 £ siga ( w ) • Dyck+(^,y). 

we&t 

We need a second notion of crossing paths, called topological intersection. Here we mean 
that, when the t-path L is drawn in the plane Q 2 there is a point x G Q 2 lying on two paths L a , 
Lb occurring in L. Because we allow rational coordinates, topological intersection is not the 
same as intersection: It is easy to give an example of a 2-path, which, when drawn in the plane 
Q 2 has one topological intersection point feQ 2 but the point x does not occur in the lists 
of points vifi, v\ } i, . . . , v\ jTl , #2,0, #2,1, ■ ■ ■ , V2 >r2 defining the 2-path. Such paths are considered 
non-crossing under our definition, even though they may have topological intersection points^]. 

We claim that there is at most one permutation w G &t such that the polynomial 
Dyck + (x w ,y) is non-zero, and that this permutation is the one we defined in Definition 12.81 
Let &( be the set of all permutations such that Dyck + (x" 1 , y ) 7^ 0, and assume that &( 
contains an element w G &{. We first make the following observation: 

(Obs) To any point v G Q 2 we associated the invariant p(v) := P2(v) £ Q/Z. The horizontal 
distance between the point x w ( a ) and the point y a is the number ri%. The vertical 
distance is the number s™ = ^n™ € Q. Because w E &( there exists a path from the 
point x w f a -\ to the point y a . Consequently is integral. This implies that p{x w c a \) = 
p(y a ) for all indices a and in particular the invariant of the point is independent 

of w g 

We show inductively that w is uniquely determined. We start with showing that the index 
w -1 (i) G {1,2, ... ,t} is determined. We claim that w -1 (i) G {1,2, ... ,t} is the minimal 
index such that the point y w -i(t) has the same invariant as xt- To see that this claim is true, 
suppose for a contradiction that it is false, i.e. assume the index u; _1 (i) is not minimal. Then 
there is an index b strictly smaller than w -1 (i) such that yt> has the same invariant as Xf 
By the observation (Obs) there exists an index a/t such that x a has the same invariant as 
xt and such that x a is connected to y^. Draw a picture (see Figure 2) to see that the paths 

^If one uses the wrong, topological notion of intersection, then the proof breaks at 8 lines below Equa- 
tion (|2.14[) : The constructed 'path' (L' c ) is not a path. 
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Figure 2. The leftmost point xt is connected to the third point y w -xm, 
and the second point x a is connected to the last point y&. Any 2-path staying 
below the line £ must self-intersect topologically. 

L a and Lt must intersect topologically. But, by construction, the invariants of x a and xt are 
the same. Therefore, any topological intersection point of the paths L a and Lt is a point of 
crossing. Thus, the paths L a and Lb are crossing. This is a contradiction, and therefore the 
claim is true. Thus the value is determined. 

We now look at the index t — 1. The point xt-\ is connected to the point y w -i(t-i)- We 
claim that w^ 1 ^ — 1) £ {1,2, . . . ,t} is the minimal index, different from w (t), such that 
y w -i(t-i) nas the same invariant as xt—i- The proof of this claim is the same as the one 
we explained for the index t. We may repeat the same argument for the remaining indices 
t — 2, t — 3, etc. Consequently w is uniquely determined by its properties, and equal to the 
permutation wq defined in Definition 12.81 

We proved that if the set &{ is non-empty, then it contains precisely one element, and 
this element is equal to wq. Therefore, if the compact trace does not vanish, then m must 
divide t or m divides h by Lemma 12.91 Inversely, assume that m divides t or m divides h. 
The permutation u)q € &t exists by Lemma 12.91 We claim that Dyck^ (i 1 " , y ) ^ 0, so that 
wo € &{. To prove this, it suffices to construct one non-crossing t-path from the points x™ to 
the points y. This is easy (see Figure 3): Let a be an index, and write rz^° for the horizontal 
distance between x%° and y a and for the vertical distance. The path L a from x™° to y a 
is defined to be the path taking — s™° horizontal eastward steps, and then diagonal 
northeastward steps. Then (L a ) is a strict non-crossing t-path and therefore Dyck^(x w °, y) 
is non-zero. This completes the proof. □ 

3. A DUAL FORMULA 

The argument for Theorem 12.101 extends to the case where h <t. This computation more 
complicated, because the permutation w € &t that contributes to Equation (I2.15P is no longer 
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Figure 3. An example of a non-crossing 4-path (L a ) in case ^ = | G Q/Z 
and t = 4. For each a, the path L a first takes n„° — horizontal steps and 
then s™° vertical steps. Note that paths with the same invariant do not inter- 
sect. 

unique and the signs £p nM m in Equation (|2.14p depend on the contributing permutations w 
(these signs are independent of w only in case h > t). We don't reproduce the computation 
here, because there is a more elegant approach using the duality of Zelevinsky. 

The Zelevinsky dual of a Speh representation with parameters (h, t) is a Speh represen- 
tation with the role of the parameters inversed, thus of type (t,h). Furthermore, taking the 
Zelevinsky dual of the formula of Tadic yields a new character formula, now in terms of duals 
of standard representations. Of course, the Zelevinsky dual of a standard representation is not 
standard, rather it is an unramified twist of products in 1Z of one dimensional representations. 
Therefore, we compute first the compact trace on the one dimensional representations, then 
use van Dijk's theorem |17^ prop. 1.5] to obtain formulas for products in 1Z of one dimensional 
representations, and finally use the dual of Tadic's formula to compute the compact traces 
on Speh representations with h < t (opposite inequality to Theorem 12. lQj) . We will then 
have computed the formula for all Speh representations. This approach seems longer but that 
is not true: The individual steps we take also appear in an equivalent form in our original 
computation. 

3.1. The trivial representation. We compute the compact traces of spherical Hecke op- 
erators acting on the trivial representation of G. We recall some definitions on roots and 
convexes from [211 §1] an d [HI Chap. 1]. 

Let P be a standard parabolic subgroup of G. Let be the center of P. We write 
e P = (_i)dim(A P /A G )_ We define ap ._ x*(A P ) ® R. If P C P' then we have A P , C A P and 

thus an induced map ap> — > ap. We write T = Ap . We define dp to be the quotient of ap 
by ap/. We write Oq = 0p and 0g = o.p . 

We write A for the set of simple roots of T occurring in the Lie algebra of Nq. For each 
root a in A we have a coroot a v in a^. We write Ap C A for the subset of a E A acting non- 
triviahy on Ap. For a G Ap C A we send the coroot a v £ (Xq to the space Op via the canonical 
surjection tig -» cip. The set of these restricted coroots a v | a G with a ranging over Ap form 
a basis of the vector space dp. By definition the set of fundamental weights {m^ G tip* | a G 
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Ap} is the basis of Op* = Hom(cip, R) dual to the basis {o^g} of coroots. Recall that we have 

the acute and obtuse Weyl chambers of G. The acute chamber cip + is the set of x G ap* such 
that (a, x) > for all roots a G Ap. The obtuse chamber + 0p is the set of x G dp such that 
we have the inequality (w^,x) > for all fundamental weights w^, associated to a £ Ap. 
We need another chamber, defined by -Op = {x G dp [ Va G Ap (w^,x) < 0}. We call this 
chamber the closed opposite obtuse Weyl chamber. Let -Tp be the characteristic function on 
ap of this chamber. Let Hm- M — > ap be the Harish- Chandra mapping, normalized such 
that |x( 77i )lp = q~( x ' HM ( m ^ for all rational characters \ of M. We define the function £^ on 
Mo = T to be the composition -Tp o (ap -» 0p () ) o Hm - 

If / G %o(G) is a function whose Satake transform is the function h € A, then we often 
abuse notation, and write £fh for the Satake transform of the function ^ f^ p °\ and similarly 
for the functions xn/ and xn! if / G %q{M). 

The following Proposition and proof are valid for any split reductive group G over a non- 
Archimedean local field. 

Proposition 3.1. Let f be a function in the Hecke algebra %q{G). The compact trace 
Tr(x<*/, 1 G ) is equal to Tr ^ fW , lp^ 2 )) . 

Proof. For comfort we prove the proposition under the additional assumption that G is its 
own derived group. We have 

Tr(^ Wp) /,l)= £ £ pTr(xiv/^,l(<5p 1/2 )). 

P=M7V 

Recall that we have the notation ipM,p £ for the Hecke matrix of a representation p of M. 
The Hecke matrix 92 i/ 2 is conjugate in M to the Hecke matrix <p -1/2 ,--1/2 = (5 -1/2 G 

™, Op l,°p °p () nM ' Pq 

T C M. Recall that the Satake transform is defined by the composition of the morphism 
/ i-> /( p °) with the obvious isomorphism %q(T) = C[X*(T)] (the Satake transformation for 
T). Therefore 

Tr(xiv/ (P) , 1(6 P 1/2 )) = 5(xiv/ (Po) )(^,-i/ 2 ). 
Using linearity of the Satake transform we obtain 

Tr( Xc G ^/,l) = l s( £ £ pxW^) (^-vO- 

\P=MN J ' P " 

Thus we have to compute the function X]p=MAf £ PXAf 011 the group T. By definition we have 

XN = Tp o Hm- 

Let W M be the rational Weyl group of T in M. Let t G T. Then 
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Thus Xiv(*) = 1 if and only if 

VaeAp: Yj ( w a,wH T {t)) > 0. 

wGW M 

We have for all a € Ap the inequality (ro^, Hr{t)) > if and only if we have (g7„ , wHrit)) > 
for all ^ € Wa/. Therefore, we have on the group T 

XN = Tp o # T . 

Thus 

X] ep XAf = I E £p? P ) ° Ht - 

P=MN \P=MN / 

By inclusion-exclusion we have 

P=MN 

This proves the proposition in case G = Gdcr- It is easy to deduce the statement from the 
case G = G& eT . □ 

Remark. Consider the space I of locally constant functions from G/Pq to C, and equip I with 
the G-action through right translations. Then, with an argument similar to the one above, 
one may compute the compact traces on the irreducible subquotients V of C. Recall from 
Borel and Walach [1] that these representations are all mutually non-isomorphic and occur 
with multiplicity one in I. Borel and Walach describe the representations V precisely; they 
are indexed by the standard parabolic subgroups of G. 

3.2. The dual formula. In this subsection we prove the dual version of Theorem 12.101 

Lemma 3.2. Let T\ = {u\ } v\), Ti = (^2,^2), = (uh,Vh) be a list of segments and 

consider the representation J := (ATi) 1 x (AT2Y x • • • x (AT/,) 1 . Then Tr(x^/ nQ , s , tt) is equal 
to q s ( n ~ s ^ 2 ~Dyck(u, v), where u a = £(u a ) and v a = £(v a + 1) for a = 1, 2, . . . , t. 

Remark. Recall that for the compact trace on the Steinberg representation, Tr(x^ fnas, Stc) 
we had the sign ep multiplied with a strict Dyck polynomial. In case n and s are coprime, 
then any Dyck polynomial from the point £(^^) to the point £(^^- + 1) is strict; consequently 
the trace on Steinberg and trivial representation differ only by the sign e p . 

Proof. The proof is the same as the proof for Lemma [2.3^ replacing the result in Equation (|2.6p 
with the result from Proposition 13.11 However, we repeat the argument for verification pur- 
poses (one has to be careful with the signs). 

Assume first that h = 1 and that it is the trivial representation of G. In the previous 
subsection we proved that 

Tr { X ?f nas , tt) = Tr (^/ffi M^ 2 )) • 
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To a monomial X = X\ X X% ■ ■ ■ X%> e C[Xf l ,Xf l , . . . X^ 1 } with e^Z and £? =1 e, = s we 
associate the graph Gx with points 

(3.1) v :=£(^), Vi :=v + (i,e 1 +e 2 + ... + ei) € Q 2 , 
for i = 1,2, ... ,n. We have X = X if and only if 

(3.2) e 1 + e 2 + --- + e l <^i, 

for all indices i < n, and (£?X = otherwise. The evaluation of X at the point 

( n—l n—3 l—Ti 
q 2 ,g 2 2 

equals the weightQ of the graph <5_y • 

— 1/2 

The trace of f nas against the representation 1t(5 p ) is equal to the evaluation of f nas 
at the point in Equation (|3.3p (use Lemma 12.21 but notice that the signs are different). The 
monomials X occurring 

S{fnas) yield paths from the point ^(^) <= Q 2 to the point ^(2=1+1). 
The condition in Equation (13. 2\i is true if and only if the graph Qx lies (non-strictly) below 
the line t. Therefore we have 



s(n—s) 

Tr( X Cf nas , 1 G ) = „ 2 Dyck(^),^ + 1)). 

. 1— n 
-x+ 



By twisting with the character v 2 as we did in Lemma 12.51 we find 

Tr ( x £7, (A(n, v)Y) = q^^ 1 By ck(£ (x), i(y + 1)), 

for all segments (u,v). Finally the argument in Lemma 12.61 may be repeated to find the 
compact traces on duals of standard representations as stated in the Lemma. □ 

Theorem 3.3. Let n be a Speh representation with parameters h,t with h <t. Let d be the 
greatest common divisor of n and s and write m for the quotient Let T a = {u a ,v a ) be 
the segments of n L . Define the points u a := £(u a ) and v a := £(y a + 1). The compact trace 
Tr(x^ fnas, tt) is non-zero if and only if m divides h or m divides t. Assume that the compact 

s{n-s) 

trace is non-zero, then it is equal to sign(wo)q 2 Dyck + (u w °, v), where the permutation 
wq G &h is defined in Definition^, 



Proof. Let ■n i be the representation dual to the representation tt. After dualizing the formula 
of Tadic for tt l we obtain an expression of the form 

(3.4) vr = signH^- 



'''Equation (|3.3|) differs from Equation (12.41) by a sign in the exponents. However, observe also that the 
graph in Equation (|3.1[) is traced in the direction opposite to the graph in Equation (|2.2[1 . 
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The involution t on ?l commutes with products. Therefore, if Xi, . . . are the Zelevinsky 
segments of the dual representation 7r\ then I L W is equal to (ATi) 1 x (AT 2 ) 1 x • • • (AT k y. By 
Lemma 13.21 we obtain 

Tr( Xc G /„ QS ,/;) = ^"-^ 2 Dyck(u«\y). 

A crucial remark is that the points u and v are all different because we assume that h < t. 
Therefore one may repeat the argument in the proof of Theorem 12.101 using the dual formula 
in Equation (|3.4p ; one only has to interchange t with h and every occurrence of the word 
"strict Dyck i-path" with "Dyck /i-path", as the paths that describe the compact traces on 
(products in 1Z of) trivial representations are not necessary strict. □ 

4. Return to Shimura varieties 

In our article [IT] we proved a formula for the basic stratum of certain Shimura varieties 
associated to unitary groups, subject to a technical condition on the Newton polygon of the 
basic stratum (that it has no non-trivial integral points). In the previous sections we have 
completely resolved the combinatorial issues that arise if you remove this condition in case 
p is totally split in the center of the division algebra. We may now essentially repeat the 
argument from [17| to obtain the description of the cohomology if there is no condition on 
the Newton polygon of the basic stratum. A large part of the argument remains the same, 
that part will only be sketched and we refer to |17| for the details. 

4.1. Notations and assumptions. Let Shx /Oe ®%>{ p ) be a Kottwitz variety [13]. Here we 
have fixed the following long list of notations and assumptions: 

(1) Let D be a division algebra over Q; 

(2) F is the center of D, assume F is a CM field of the form F = KF + C Q, where F + 
is totally real, and /C/Q is quadratic imaginary; 

(3) * is an anti-involution on D inducing complex conjugation on F; 

(4) n € Z>o is such that dim^D) = n 2 ; 

(5) G is the Q-group with G{R) = {x € D^\g*g G i? x } for every commutative Q-algebra 
R; 

(6) h is an algebra morphism h: C — » Dr such that h(z)* = h(z) for all z € C; 

(7) the involution x \-t /i(z) _1 x*/i(i) on Dr is positive; 

(8) X is the G(R) conjugacy class of the restriction of h to C x C C; 

(9) /i£l,(G) is the restriction of h C : C x x C x -> G(C) to the factor C x of C x x C x 
indexed by the identity isomorphism C — > C; 

(10) E C Q is the reflex field of this Shimura datum (G,X, h~ 1 ); 

(11) £ is an (any) irreducible algebraic representation over Q of Gq; 

(12) Let /oo be a function at infinity having its stable orbital integrals prescribed by the 
identities of Kottwitz in [12]; it can be taken to be (essentially) an Euler-poincare 
function [T3], Lemma 3.2] (cf. [9]). The function has the following property: Let 
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7TQO be an (g, X 00 )-module occurring as the component at infinity of an automorphic 
representation ir of G. Then the trace of against tt^ is equal to the Euler-Poincare 
characteristic X^£o ^oo(— 1)* dimFP(g, K^; -k^ (8> £), where is a certain explicit 
constant (cf. [13, p. 657, Lemma 3.2]). 

(13) p is a prime number where Shx has good reduction [141 § 5], and we assume that p is 
split in /C/Q; 

(14) K C G(Af) is a compact open subgroup, small enough that Sh^ /Oe ® ^p is smooth 
and such that K decomposes as K p K p where K v is a compact open subgroup of G(A?) 
and i£p is a hyperspecial compact open subgroup of G(Q P ). 

(15) z/ p : Q — )• Q p is a fixed embedding, : Q — > C is another fixed embedding, the fields 
F, F + ,E, /C are all embedded into C; 

(16) p is the E'-prime induced by v p \ 

(17) F q is the residue field of E at the prime p and ¥ g is the residue field of Q at u p ; for 
every positive integer a, Ep >a C Q p is the unramified extension of E p of degree a; ¥ q a 
is the residue field of Ep ;a ; 

(18) f.B^f Shjc, F , is the basic stratum [21] (cf. [TOMESE)] ) ; 

(19) is the characteristic function on G(Q P ) of the subset of compact elements (cf. [7]); 

(20) I is a prime number and an algebraic closure of Qi together with an embedding 

Q c Qi, 

(21) C is the ^-adic local system on Sh^- /Oe <S>^(p) associated to the representation £® 
of % [Hip. 393]; 

(22) U C G is the subgroup of elements with trivial factor of similitudes; 

(23) for each infinite F + -place v, the number s v is the unique integer < s v < \n such 
that U(R) ^]J v U(s v ,n- s v ); 

(24) the embedding Q C Q p induces an action of the group Gal(Q p /Qp) on the set of infinite 
F+-places. For each Gal(Q p /Qp)-orbit p we define the number s p := ^2 v< = p s v , and 
we write a p for the partition (s v ) v&p of the number s p ; 

(25) the function f a is the function of Kottwitz [12] associated to \x (cf. [T71 Prop. 3.3]). 

Remark. The second condition (2) is particular for our arguments, and does not occur in |13j . 

4.2. The main argument. In this article we compute the factors Tr(xc^ P ^ fa, ftp) occurring 
in Theorem 14.31 below. We need to introduce two classes of representations: 

Definition 4.1. Consider the general linear group G n over a non-Archimedean local field. 
Then a representation ir of G n is called a (semistable) rigid representation if it is equal to a 
product of the form 



Y\ Speh(x a ,y)(e a ) € 11, 



a=l 



where y is a divisor of n and (x a ) is a composition of ~, and e a are unramified unitary 
characters. 
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Definition 4.2. A representation tt of the group G(Q p ) = Q* x n p | p GL n (F+) is called a 
rigid representation if for each .F + -place p above p the component ir p is a (semistable) rigid 
representation of GL n (i ? p f ) in the previous sense: 

k 

7T P = IJ Speh(x Pja , y p ){e p , a ) € ft, 

0=1 

where two additional conditions hold: (1) y p = y p i for all p,p'\p, and (2) the factor of 
similitudes Q* of G(Q P ) acts through an unramified character on the space of tt. We write 
y := y p and call the set of data (x P:a ,e P:a ,y) the parameters of tt. 

Remark. Recall that we work in the semistable setting, both notions of rigid representations 
that we introduced above in the semistable setting also have a natural variant in the non- 
semistable case. 

Theorem 4.3. Let a be a positive integer. Assume the conditions (l)-(25) from §5.i. Then 

oo 

(4.1) ^(-l) 4 Tr(/^x ^,Blt(B fq , C C)) = £ Tr( X ff a , tt p ) • Tr(f, ^ ). 

1=0 7rC.A(G) 

7T p rigid 

Remark. Using recent results obtained with Lapid [19] it is possible to extend the above 
theorem to the other Newton strata. However the result will be combinatorially complicated. 
We hope to include this result soon. 

Proof of Theorem \4-3\ Write T(f p , a) for the left hand side of Equation (|4.ip . By Proposition 
3.4 of [TT] we have 

(4.2) T(f, a) = Tr( X ° UfaF, A(G)), 

for all sufficiently large integers a. To simplify notations, we write / := foofaf p - 

Let 7T C A(G) be an automorphic representation of G contributing to the trace 
Tr(xff,A(G)). In [loc. cit, p. 20] we explained that ix may be base changed to an automor- 
phic representation BC(tt) of the algebraic group /C x x D x , and that, in turn, BC(tt) may be 
send to an automorphic representation II := JL(BC(tt)) of the Q-group G + = IC X x GL n (F). 

The representation II is a discrete automorphic representation of the group G + (A), and II 
is semistable at p. The classification of Moeglin-Waldspurger implies that tt p is the irreducible 
quotient of the induced representation 

t ,GL n (A F ) / | | 

Ind p(A^) • I 2 ,'~M- \ 2 ) > 

where P C GL n is the homogeneous standard parabolic subgroup having y blocks, and each 
block is of size n/y; the inducing representation uj is a cuspidal automorphic representation 
of GL n / y (k F ). 

The representation II comes from an automorphic representation of the group G via Jacquet 
Langlands and base change. Therefore, II is cohomological and conjugate self dual. These 
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properties descend, up to twist by a character, to the representation u. The Ramanujan 
conjecture is proved to be true for the representation u by the articles [5l[8l[27j- Thus the 
components u) v of u are tempered representations. Note that, of course, the components n„ 
are not tempered if IT is not cuspidal. 

An easy computation shows that 7r p is a rigid representation for all .F + -places p dividing 
p \17\ Thm. 2.1]. This means that there exists a positive divisor y of n, a composition 
^ = X^a=i x ai an d unramified unitary characters e a such that 

(4.3) tt p ^ Ind^;|^ +) (g) Speh(x a , y)(e a ), 

a=l 

where P C GL n is the standard parabolic subgroup corresponding to the composition (x a y) 
of n, and the tensor product is along the blocks of the standard Levi factor M of P. In 
Equation (|4,3p the number y is of global nature and does not depend on p. The other data, 
k, (x a ) and e a do depend on the place p. □ 

We work under the condition that p is split in the center F of the algebra D. Because the 
prime p is completely split in the extension F/Q we have by [17\ Prop. 3.3] that 

fa = l 9 -a ® (g) fnas^ G Wo(G(Q„)), 

wGHom(F+,R) 

where the numbers s„ are the signatures of the unitary group (cf. subsection 1). We compute 
= II Tr ( X c L " {Qp) fn aSv ,Ind^ p) (g) Speh(x„,j/)(e„, a ) ) 

?;eHom(F+,IR) V a=l / 



_[1 II e ^ a ^" 

v «eHom(F+,R) a=l 



r 



«GHom(F+,R) V a=l / 

Write C% € C for the product Ela^C^ Sv n The polynomial 

(4.4) Tr |xc L " (Fp+) U P ' Ind P(4f } ® Speh{x a , y)j G C[q% 

is computed in Theorems 12.101 and 13.31 to be a polynomial defined by the weights of certain 
non-intersecting lattice paths. In particular the trace in Equation (|4.4p vanishes unless the 
number 



(4.5) m t 



gcd (y ■ x p>a , 1L ^ L s p ) gcd(n, s p ) 
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is an integer, and divides either x p>a or y. We make the assumption that the compact trace 
Tr(x^/a) Kp) is non-zero and therefore these divisibility relations are satisfied. 

The number (" £ C is determined by the central character co n : Z(A) — > C x of the auto- 
morphic representation ir via the Equation: 

r 

s p,a \ Q P 

0=1 



(4-6) ^(xT s/n = e s (q a ) • J] II ^ U~ S "~ Q ) = C 



pip 

where e s is the contribution of the factor of similitudes, and x G Z(A) is the following element 
of the center Z of G: 

x : = (1) x [q, (q p ) p ] G Z(A') X [<Q* x F+ p X ] = Z(A). 

The divisibility relations in Equation (|4.5[) assure that taking the rational power s/n of (^(x) 
on the left hand side makes sense. 

Remark. The number £ n is a Weil-g- number of weight determined by the local system £, 
cf. [17, Eq. (3.10)]. 

Definition 4.4. We call a rigid representation tt p of G(Q P ) of B-type if for all p, gcd(n, s p ) 
divides the product y ■ x p>a . Furthermore, for each F + -prime p and each index a, the number 
m Pt a divides either y or x p>a . 

We have proved that only the -B-type representations contribute to the (alternating sum 
of the cohomology spaces) of B. Let ir p be a -B-type representation of G(Q P ). Then we write 

Po^^Tr^V^^GC^]. 

We computed this polynomial in the first 4 sections of this article. Explicitely, it is the product 
over all p, over all indices a of the polynomial 

Sp(n—s p ) 

(4.7) e-q 2 a ■ Dyck+ (f° , y) , 

where the lists of points x, y G Q 2 are defined by: 

(1) If x a < y, then x, y are of length x a , and for each b we have 

o( x a-y\ , Jxa + y 

Xb'-=z[ — ^ — , and y b : 



2 J V 2 

(2) if x a > y, then x, y are of length y, and for each b we have 

e(y- x a\ , „{x a + y 

x b '■= * — ^ — > and 2/b : = 



where £ C Q 2 is the line of slope ^ going through the origin. The notation does not show, 
but the points x, y and the permutation wq depend on p. The symbol wq is a permutation 
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in the group & m i n ( Xa ,y) an d is determined by Definition 12,81 The symbol e in Equation (|4.7p 
is a sign and is equal to 

(4.8) v • sign(wo), 

where the sign v is equal to (— \y n ~ Xa \{ x a <n and it is equal to 1 otherwise. 

4.3. Application: A dimension formula. In our previous article [17] we explained that 
Formula (|4.ip gives a formula for the number of points in B if one takes f p = Ikp and C 
equal to the trivial local system. Using this simplified formula we proved in [17, Prop. 4.2] a 
dimension formula for the basic stratum. We now extend this result to the Shimura varieties 
satisfying conditions (l)-(25) from the first subsection, with p completely split in F + . 

Take f' p = Ikp and C in Theorem 14.31 so that the right hand side of Equation (|4.ip counts 
the number of points in B over finite fields. We computed the class of representations at p 
contributing to this formula. Each representation tt p at p contributes with a certain function 
P{q a ) to the zeta function of B. We call the order of tt v the order of the function P(q a ) (as 
function in q a ). 

Proposition 4.5. The trivial representation tt p = 1 contributes with the largest order to the 
right hand side of Equation (14.11) . 

Remark. In the statement of the proposition, we mean 'largest order' in the non-strict sense. 
In general there are multiple representations contributing to the formula with the same order. 

The order of the trivial representation is easily computed, it is equal to: 

(4-9) EfE f2 ^ + Erif 1 V 

p\p Vep j=o p / 

(cf. [la (4.4)]). 

Proof of Proposition Let ir p be a unitary rigid representation. Pick one p\p. The com- 
ponent 7T P is a rigid representation of G p := GL n (F£). 

Assume first that ir p is a Speh representation. We assume that h < t, so we will work in the 
dual setting. Treatment of the non-dual case is essentially the same (see Eq. f|4.25j) at the end 
of this argument below). Let T\ = (u%,vi), T2 = (^2,^2), ■ ■ ■, Th = {uh,Vh) be the segments 

Q 

of the Zelevinsky dual ir p of 7r p . By Tadic's formula the compact trace Tr(x c P fp-, %>) is 
an alternating sum of compact traces Tr(x c p fp,Pw) on Zelevinsky duals of certain standard 

Q 

representations I w . The traces Tr(x c p f p ,I^) can be described using graphs as we explained 
in the first section of this article. The intuition is that the closer the graph is to the line £, the 
larger its weight is, and we claim that the largest weight is attained by trivial representation. 
More precisely, we claim that for all / £ "Hq(G p ) and for all permutations w G &h we have 

(4.10) Ord(Tr( X ? p /,!*)) < Ord(Tr( X ^/, l Gp )), 
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where with Ord(/i) S Q of an element h G A + we mean the largest element x G Q such that 
q x occurs as a monomial in the expression of h with non-zero coefficient. By Proposition 13.11 
and Dijk's integration formula for compact traces we have 

Tt( X ^f,I^ = q S± ^ 1 ■ Yl c x -Q x (u w ,v)eA+, 

where X ranges over the monomials X G A of the Satake transform S(f) of /, cx G C is 
their coefficient and where we should explain the notation Qx{^ ,v). The symbol u denotes 
the list of points u a := i(u a ) G Q 2 for o = 1, . . . h and the list of points v is defined by 
v a := £(v a + 1) G Q 2 . The symbol Qx is the graph of the monomial X as defined in the first 
section. Recall however that Qx is only well-defined up to the definition of its starting point. 
The representation I L W is obtained by induction from a one dimensional representation of a 
standard Levi subgroup M w of G p . Let (n™) be the corresponding composition of n, and let 
k w be the length of this composition. We cut the graph Qx into k w pieces, the first piece 
contains the first nf steps of Qx, the second piece contains the next block of n™ steps of Qx 
and so on. Thus instead of one graph Qx we now have k w graphs, Q x 1 , Q X2 , ■ ■ ■ 5 Qx k > au 
well defined up to their starting points. We let the starting point of the graph Q X \ be iTf, 
the starting point of the graph Q x 2 is by definition , and so on. Then Q x 1 , Q x 2 , . . . are 
well defined graphs in Q 2 , and due to our definition of starting points, we have 

(4.11) [] weight^) = Tr (x, (I^ No (5^ /2 )) G A+. 

a=l V 

G G 

The condition £ c p Xm w X /Oonl means precisely that the graphs Q x a have endpoint equal 
to v a and that these graphs do not cross, but may touch, the line i. 

Starting from the monomial X we can also defined a second graph Hx, such that 

weight(^ x ) = Tr (x, 1(<5 P() 1/2 )) € A+ . 

This graph has starting point x = ^(^i 2 ) and end point y = £(^y~ + 1); the steps of 7-Lx are 
defined by Formula (12. 2j) . 
We now claim that 

h 

(4.12) 11 weight^J < weight(^ x ) G A + , 

a=l 

for the obvious meaning of '<'. 

Before we prove the claim, let us first show a simple fact of graphs. Let Q be any graph in 
Q 2 . Then we have, for any point (a, b) G Q 2 that 

(4.13) Ord(£ + (x, y)) = Ord(£) - x ■ Height(£), 

where the height of Q, Height(^), is the vertical distance between the initial point of Q and 
its end point. This formula is easily seen to be true: The order Ord(^) is equal to the sum 
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of —a ■ e over all diagonal steps (a, b) — > (a + 1, b + e) occurring in the graph Q. Adding the 
point (x, y) to Q amounts to changing —a ■ e to —(a + x)e in the definition of the order of Q. 
Thus the order of Q is shifted by the sum, over all diagonal steps (a, 6) — > (a + 1,6 + e), of 
the value — xe. This gives the formula in Equation (|4,13p . 

We now return to the graphs Qx and %x introduced earlier. We cut Hx into h consecutive 
graphs. The first graph Hx,i consists of the first rif steps of Hx, the second graph 1-Lx,2 
consists of the second block of ri^ steps of Hx-, and so on. The graphs Qx,a have the same 
shape as the graphs l~Lx,ai but they are shifted (the graphs are constructed starting from the 
same monomial X). Therefore we have the relations: 

(4.14) (Vo): Hx,a = Gx, a -£(u wi a)) + K k T k + < + •••<-!)> 

(we subtract the initial point of Gx,a, and then add the initial point of 7ix,a)] in the above 
formula we have the convention that 



n? + nf + ... + n™_ 1 = 0, 

in case a = 1. Note also that 



Ord(H x ) = J20rd(n x ,, 



a=l 



and similarly for Qx- By Equations (|4.13p and ()4.14p we have 

OrdCHx.a) = Old{G X ,a) ~ U w(a) . S « + (1=2 +„« + ... + <L X ) • C 

where := n™ ■ = Height(^x,a) = Height(^x,a)- Thus we have to compute the following 
expression 

(4.15) C(W) = -J2 + < + ^ + ■■■ + <-! ~ U v(a)) <. 

n a=l ^ ' 

To show that Equation (14.12p is true, we show that C(w) < for all permutations w. 

To prove that C(w) < 0, we may ignore the factor f t in the above expression. We prove in 
two steps that C(w) < for all w. We first determine the permutation w such that the value 
C(w) is maximal (Step 1). Then we compute for this particular permutation the value C(w), 
and observe that it is non-positive (Step 2). 

We begin with Step 1. We want to determine w such that C(w) is maximal. Let us first 
simplify the expression somewhat. The expression C(w) is maximal for w if and only if 

h 

(4.16) + + + n *_ x - Uw(a) ) 

0=1 
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is maximal. To derive (|4.16p we useqj, that the sum 5Z =1 !L 2^ n a equals nKp- and therefore 
this sum does not depend on w. (Similar arguments will appear also below.) We have 

(4.17) < = (*±* - a) - ^1 - (w(a) - 1)) + 1 = t - a + w(a). 
and 

(4.18) u w(a) = l — (w(a) - 1). 

We plug Equations (|4.17p and (|4.18p into Equation (j4.16j) to get 

^((t-l+ W (l)) + ... + (t-(a-l)+ W ( fl -l))-^ + Wa)-l)) < 

a=l ^ ' 

As before, this expression is maximal for w, if and only if the expression 
h 

(4.19) 53 ( w 0) ~ 1 + u '( 2 ) - 2 + . . . + w(o - 1) - (o - 1) + w(a)) (t - a + u>(a)) 

a=l 

is maximal. Equation ()4.19j) is maximal for w if and only if the expression 
h 

(4.20) 53 M 1 ) ~ 1 + w ( 2 ) - 2 + . . . + w(a - 1) - (a - 1) + w(a)) (w(a) — a) 

0=1 

is maximal. We may rewrite (14.20P to 

h h 

(4.21) + w(2) + • • • + u>(a))a - 5^(1 + 2 + . . . + (a - 1)M°) 

a=l a=l 

We rearrange the first sum as follows. Count for each index a the coefficient of w{a) to get 
h h 
53 O(l) + «>( 2 ) + • • • + m(a))a = 53 + 1 ~ a)w(a), 

a=l a=l 

where 

p(a) := 1 + 2 + 3 + .. . + a = ±a(a + 1). 

Thus (ET2Tj) equals 



53(p(t + 1 - a) - p{fl - 1) Ma) 

a=l 

The function v{a) defined by u(a) = p(h + 1 — a) — p(a — 1), is strictly decreasing in a because 

via + 1) - f (a) = -(h + 1). 

We are looking for w such that 

h 

53 "(«) 



a=l 



See Equation (|2.1|l . but note that by duality the roles of h and t are switched. 
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is maximal, with v{a) a strictly decreasing function for a £ {1, 2, . . . , h}. This maximum is 
attained by the permutation w defined by a i-» h + 1 — a. This completes Step 1. 

We now do Step 2. Thus we have w(a) = h + 1 — a for all indices a G {1, 2, . . . , /i}. We 
compute the sum 

h 

. / I — T? \ 

* I ' " 7/) ?/i . 7/) \ 



(4.22) = £ + nf + < + ... + nJLi - ^ (a) ) 



"a ' 



a=l 

We have 



n« = Va - u w{a) + 1 = (^^- - oj - ~ ~ ^) + 1 

(4.23) = i - a + iu(a) = t- a+(/i + l)-a = t + /i + l-2a, 
and we have 

(4.24) u w(a) = - (w(a) - 1) = ^ -(h- a). 



E< = X;* + ^+l-2a. 



Note also that, 

n = 

a=l a=l 

(cf. Equation (|2.1 j) ) . Thus, Equation (|4.22p becomes 

E(^+(& + /i+1 - 2fc ) -(^- (/l " a) )) ' (t + /l+1 " 2a) 

An easy (but somewhat lengthy) computation shows that this last formula simplifies to 
n(h — t). By assumption we have h < t. We conclude that the value in Equation (|4.22p 
is non-positive, which is what we wanted to show. We have now established the claim in 
Equation 

With the same proof, but using the non-dual instead, one can show that 
(4.25) Ord(Tr( X ^/,/ w )) < Ord(Tr( X ^/, St Gp )), 

is true for all representation I w occurring in Tadic's formula for Speh representations tt with 
h > t. Because 

Ord(Tr( X ^/,St G J) < Ord(Tr( X ^/, l Gp )), 
the inequality of Equation (|4.1Up is true for all Speh representations. We leave it to the reader 
to deduce that Equation ()4.10p also holds for products of Speh representations, and also for 
the rigid representations of G p (with the characters e a trivial). 

We return to the group G(Q P ) and the full representation ir p . The compact trace 
, 7T P ) is the product of the traces on the components, 

Tr(xc P 'f a , TTp) = Tr Q x (/ s , vr s ) • Ti(xc P f p , n p ), 

p\p 
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(the first term in the product is the contribution of the factor of similitudes). We proved that 
all the terms of this product are bounded by the trace on the trivial representation. This is 
then also true for the entire product. □ 

We now deduce a formula for the dimension. 
Theorem 4.6. The dimension of the basic stratum B is equal to 

p\p \ V ^P j=0 f 

Proof. Apply Proposition 14.51 and Theorem 14.31 to find 

p\p \v£(p j=0 P 

We now prove the opposite inequality. We return to the final formula we found in Theorem l4.3l 

oo 

(4.26) ^(-l) i Tr(f°°Px$$,Bi t (B ¥q ,L*C))= £ Trtf>f a , %) . Tr(/*>, ). 

1=0 tt<zA(G) 

TTp rigid 

We take in this formula f p and C of the following form. Let p\ be a prime number with 

• pi is different from £,p; 

• the group G splits over Q pi ; 

• the group K splits into a product K pi K Pl of a hyperspecial group at p\ and a compact 
open subgroup K Pl C G(A Pl ) outside p\. 

We take 

• f p ^ = \ K vvr- 

• f pi is an arbitrary K pi -spherical function; 

• C = (the trivial local system). 

There exist only a finite number of representations tt Pi contributing to Equation (|4.26p . and 
one of these representations is the trivial representation. Thus we may find a spherical Hecke 
operator f Pl € 'H(G(Q Pl )) such that 



1 ^pi - iccQpj 

otherwise, 



for all representations tt Pi occurring in Equation (|4.26p . We consider the Hecke operator 
f p := Ikppi ® f Pl in Equation (|4,26p . By construction, any automorphic representation 
vr C A{G) contributing to Equation (|4.26p has 7r^ 1 = 1g(Q p )■ By a strong approximation 
argument, the representation tt is one dimensional, and in particular Abelian. Consequently, 
at the prime p ^ Pi, the representation tt p is a twist of 1g(q p ) by an unramified character \p- 



See for example Lemma 3.6 in our article [T7], although this result is of course well known. 
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Because the representation £ at infinity is trivial, the character \ p is of finite order. Therefore 
there exists an integer r > such that, whenever r divides a, we have 

TV( Xc GWp) /a,vr p ) = Tt(x? iQp) f a , 1), 

for all representations tt p contributing to Equation (|4.26p . From now on we consider only a 
such that r\a. The right hand side of Equation (|4.26p simplifies to 

where C is some non-zero constant. Thus for our choice of f°° p the trace 

oo 

(4.27) £(-1)* x Bi t (B Wq Me)) 

i=0 

grows with the order of the trivial representation. View YliZo(~ l) 1 -^^! > ^) as a virtual 
representation of the group (^p) z , and write it as a linear combination of the characters of this 
group. The character of highest order occurring in this expression determines the dimension 
of the variety B. By the conclusion in Equation (|4.27|) there occurs a character whose order 
is at least Ord(Tr(x^ < ^ P Va; !))■ This means that 

p\p \v£fp j=0 P J 

This completes the proof of the Theorem. □ 

Remark. The above formula confirms the conjecture for the dimension of the basic stratum 
specialized to the cases we consider. See for example [16]. 

4.4. Application: Vanishing of the cohomology. In our previous article |17| we assumed 
that the signatures s p are coprime to the number n. Under these conditions the cohomology 
of the basic stratum is very simple: Locally at the prime p, only the trivial representation 
and (essentially) the Steinberg representation contribute to Expression (|4.ip . In fact this is 
true in a larger class of cases: 

Corollary 4.7. Assume there is one F + -place p above p such that s p is coprime to n. Then 
only the Steinberg representation and the trivial representation contribute to the formula in 
Equation (j4.1[) . 

Proof. This follows directly form the definition of rigid representation of the group G(Q P ). □ 

Remark. In the article [T7] we assumed that, for all p, the number s p is coprime to n or s p 
is equal to or n. Only under this larger assumption the compact trace on the Steinberg 
representation coincides with the compact trace on the trivial representation (up to sign), 
just as in [17]. In the above Corollary this need not be the case. 
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4.5. Application: Euler-Poincare characteristics. Finally we have a remark on the 
Euler-Poincare characteristic of the variety B. The evaluation at q = 1 of our formula gives 
the expression of the Euler-Poincare characteristic. Thus to compute the Euler-Poincare 
characteristic we get the combinatorial problem to compute, apart from dimensions of spaces 
of automorphic forms, the number of non-intersecting Dyck paths. This problem has been 
considered in an equivalent forms in the literature; a good starting point are the books of 
Stanley [28] and the references therein. 

5. Examples 

We end this article with some examples. Let us first explain why we need the condition 
that p splits completely in the center of D. 

5.1. Products of simple Kottwitz functions. To study the reduction modulo p of unitary 
Shimura varieties, the simple Kottwitz functions f nas as we defined them in Equation (jl.ip 
are not enough. These functions count only points of unitary Shimura varieties if the group G 
of the Shimura datum is of the following kind. Consider a unitary Shimura variety associated 
to a division algebra D as in the previous section. Let U in G be the subgroup of elements 
whose factor of similitudes is equal to one. Then U is a unitary group and U(M.) is isomorphic 
to a product of standard unitary groups U(p T ,q r ) with r ranging over the infinite places of 
the maximal totally real subfield F + of the center F of D. The function f nas counts points 
on the reduction of Sh^ modulo p if we have p T = or q T = for all -F + -places r, but with 
one F + -place excluded. For the excluded F + -place to we must have p TQ = s or p TQ = n — s. 
For unitary Shimura varieties with several non-zero signatures at infinity, one will need to 
consider products of the functions f nas for several different values of s. 

Remark. Compact traces do not commute with products of Hecke operators. 

Example. Let us assume that there are two infinite -F + -places tq,t\ with p TQ = p T1 = 1 and 
that p T = for all other r. Choose embeddings CdQc Q p , so the group Gal(Q p /Q p ) acts 
on the set of infinite places of F + . Assume the places to and t\ lie in the same Gal(Q p /Q p )- 
orbit and assume a is sufficiently divisible such that the S Pja -algebra F + ®Ep t0l is split. Then 
the function counting points in the set #Sh/^(F g Q) is (essentially) the convolution product 
/ = fnai * fnal £ Ho(GL n (F)) , where F is some finite extension of Q p . An easy computation 
shows that / = 2q a f na2 + f n (2a)i, and therefore Tr{ X ^f, 1 G ) = 2(g + q 1 + . . . + q a[ ^ i ) + 1 (use 
the Example in \17\ p. 14]). Consequently the number of points in the basic stratum over the 
field F q a is the product of the above polynomial times a cohomological expression depending 
only on the class of the degree a in the group Z//iZ, where h is related to the class number of 
the oocenter of G [TTJ Cor. 4.1]. In particular the variety is of dimension [§J in this case. If 
we assume instead that To and t\ lied in a different Gal(Q p /Q p )-orbit, then the basic stratum 
of Shft- is a finite variety. Whether or not To and t\ he in the same Gal(Q p /Q p )-orbit is a 
condition on how the prime p decomposes as a product of prime ideals in the ring of integers 
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Op+ of F + . Thus, roughly speaking!^, the form of the function a i— > Tr(xf^ p ^f, 1g(Q p )) 
depends only on two pieces of information: (1) The signatures of the unitary group at infinity, 
and (2) how the prime p decomposes in F + . 

5.2. Two different prime factors. Assume F + is of degree 2 over Q and n is a product 
of two primes x, y with x < y. Let U C G be the subgroup of elements whose factor of 
similitudes is trivial. We assume U (R) is isomorphic to U(x, n — x)(R) x U(y,n — y)(R). The 
reflex field E of the Shimura datum coincides with the field F. 

There are two cases to consider, either the prime p where we reduce Shx splits in F + or p is 
inert (but unramified). Assume that p splits, then G(Q P ) = Q* x GL n (Q p ) x GL„(Q P ). Recall 
that we picked an embedding z^ p : CQ — y C^p* Therefore the factors of the product GL ri (Qp) x 
GL n (Q p ) are ordered: the embbeding u p identifies the two F + -places t\,T2 at infinity with the 
two .F + -places pi, p2 above p. Via the isomorphism U(M) = U(x, n — x)(M) x U(y, n — y)(M) 
we associate to Ti,T2 (and thus to pi, a signature equal to x or y. Assume that pi (and 
7~i) correspond to x and p2 (and T2) correspond to y. Similarly, the first factor of the group 
GL„,(Q P ) x GL n (Q p ) corresponds to pi and the second factor corresponds to p2. 

The i3-type representations of G(Q P ) are the representations contributing to the cohomol- 
ogy of the basic stratum. Ignoring the factor of similitudes, the -B-type representations of 
G(Q P )/Q p x = GL n (Q p ) x GL n (Q p ) are: 

k 

(5.1) Speh(x, y)(e) ® Y[ Speh(x a , y)(e a ), 

a=l 

k 

(5.2) Yl S P eh (ya, x)(e a ) 8) Speh(a;, y)(e), 

a=l 

(5.3) St G (e)®Sto(e') 5 

(5.4) l G (e) ® l G [e'), 

where, in these equations the number k can, a priori, be any positive number. In Equa- 
tion (|5,ip . the symbol (x a ) ranges over the compositions of the prime x and in Equation (|5.2p . 
the symbol (y a ) ranges over the compositions of the prime y. Xlie symbols £j£^s a denote 
arbitrary, unrelated, unramified unitary characters. 

5.3. Some explicit polynomials. We specialize our first example further, and assume that 
x = 2 and y = 3, so U(M) = U(2,A)(R) x f7(3,3)(R). We write down the polynomials 
Tr(x^ fa, ftp) £ A + for the representations that occur. The unramified characters £, £ a , e 1 and 
the factor of similitudes have no influence on the form of the polynomials, so we leave them 
out. 



'To obtain the precise statement use the discussion at Equation (3.1) of [loc. tit]. 
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The computation of the compact traces on the representations 7r p = Speh(3, 2) and tt p = 
Speh(2, 3) is done in the Figures 4 and 5. Recall that the computuation on Speh(3, 2) is 
done via the segments of its dual representation Speh(2,3). The Zelevinsky segments of the 
representation Speh(2, 3) are { — 5,5,!} and { — |, — 5,5}- To compute the compact traces 
we consider the line I in Q 2 of slope f- and consider the weights of non-crossing lattice paths. 
In our case there are two possible slopes, slope \ and slope |; these yield several different 
polynomials. 



Figure 4. The compact trace on the representation Speh(3, 2) with respect to 
the function fe a 3- We have ^ = \, and x\ = (-{—\), £2 = (■(— §), Vi = ^(| + 1) 
and y2 = (-{\ + 1). The permutation wq is equal to (12). We see that there are 
two Dyck 2-paths going from the points x 1 " to the points y, and one of those 
paths is non-strict because it touches the line I. Therefore Dyck+(x u, °,y) = 

g -l/2a-l/2a-3/2a = g -5/2a anG l Dyck+ (x^' , y ) = q~^/ 2a + q~^l 2a . We conc l u d e : 

Tr(x c G(Qp) /6a3,Speh(3,2)) = (-l^sign^^ V 5/2 ° = ~q 2a - 



Figure 5. The compact trace on the representation tt p = Speh(3,2) with 
respect to the function /6«2- We have ^ = 5, and X\ = £2 = £{— §), 

Vi = + an d V2 = ^(5 + !)- The permutation wq is the trivial permutation. 
There is one Dyck 2-path going from the points x 1 " to the points y and this 2- 
path is strict. Therefore Dyck+ (x"' 1 , y) = Dyck+ (x 1 " , y ) = g -i/2a-3/2a = g -a 
We conclude: Tr( X c (Qp) f 6a 2, Speh(3, 2)) = (-lf-'sign^g 1 ^ = -q 3a . 
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In the illustrations we found that 

Tr( Xc GWj,) /6 a 3,Speh(3,2)) = -g- 2a 
Tr( Xc GWp) /6 a 2,Speh(3,2)) = -g 3a . 
Using the duality and the computation in the figures, we find that 



Tr( X r VVp) /6a3, Speh(2, 3)) = q 2a + q 3a 
Tr( X ? (Qp) fca2,Speh(2,3))=q 2a . 
By drawing the picture, we see in a similar manner to the illustrations that 
(5.5) Tr( Xc G V4a2,Speh(2,2))=g 3Q . 
and 

Tr(x?'f6«2,St G6 ) = -(q a + q 2a ) 
Tr(x c G6 / 6 a3, l Ge ) = 1 + Q a + 2q 2a + q 3a 
(5-6) Tr( Xc G «/ 6a3 ,St G6 ) = -(l + ^). 

The representations at p occurring in the alternating sum of the cohomology of the basic 
stratum are (up to twists): 

Speh(2, 3) ® Speh(2, 3), Speh(2, 3) ® (l Gs x l Gs ); 

Speh(3, 2) <g> Speh(3, 2), (Speh(2, 2) xl G2 )® Speh(3, 2) (l Ga x Speh(2, 2)) ® Speh(3, 2), 
(l Ga x l Ga x 1 G2 ) ®Speh(3,2); 

St Ge ®St Ge ; 
1g 6 ® 1g 6 - 

Let us ignore the factor of similitudes of the group G(Q P ). On the group GL2(Q P ) xGL2(Q p ) 
the function of Kottwitz is equal to fe a 2 <8> fea3 € %o(GL n (Q p )) (g) ^o(GL ra (Q p )). With the 
formulas we gave above the compact traces on the representations in this list are now all 
explicit. 
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